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Abstract — In this work, we define a new pilgrims face recog- 
nition and face detection dataset, called Hajj and Umrah facial 
dataset. The new developed dataset presents various pilgrims' 
images taken from outside the Holy Masjid El-Harram in 
Makkah during the 2011-2012 Hajj and Umrah seasons. Such 
dataset will be used to test our developed facial recognition 
and detection algorithms, as well as in the missing and found 
recognition system (2). 



I. Introduction 

Every year millions of muslims arrive to perform the 
holy rituals of Hajj and Umarh in Makkah Al-Mokarramah, 
Kingdom of Saudi Arabia. During these rituals the Saudi 
authorities save no effort to facilitate the stay of the pilgrims 
in the kingdom. The greatest challenge is the huge number 
of missing persons and unidentified deaths every year. So, 
an efficient monitoring system is essential to provide means 
of tracking missing and found individuals and identifying 
them in order to take the necessary actions. To solve this 
problem, we describe a new pilgrims face recognition dataset, 
HUFRD, which will be used in the CrowdSensing system for 
recognizing missing and found people [2], see sample classes 
of HUFRD in Fig. [Q 

Facail recognition is one of the most complex applications 
in the field of Computer Vision. Developing a computational 
model of human faces for detection and recognition, though 
interesting, can prove to be a very challenging task. The rapid 
progress in PCs' speed certainly assists the operation of face 
detection and recognition to be done in real-time. 

One of the most crucial steps in the CrowdSensing system 
that we describe in [2] is to how to detect the individual faces 
in an image and run a face recognition algorithm through 
a database of registered pilgrims. Therefore, the proposed 
approach is more focused on developing a pattern detection 
algorithm that does not depend on the three-dimensional 
complex data of the face, but it depends on the general outer 
Silhouette that is almost shared between all faces. Our goal 
is to develop a computational model for face detection and 
recognition that is reasonably simple, and acceptably accurate 
under various conditions of lighting, facial expressions, and 
background environment. 

In general, facial dataset type has a strong effect in face 
recognition performance, it contains multiple images for each 
person. Facial datasets depend on several different factors such 
as: 

• Facial expression such as sadness, happiness, and facial 
pose. 
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Fig. 1: The new Hajj and Umrah facial recognition 
dataset presents various images taken outside El-Harram in 
Makkah 2011-2012 Hajj and Umrah seasons. 



Occlusion: faces may be partially occluded by other 
objects (like wearing glasses ). 

Imaging conditions like lighting and camera resolution. 
Presence or absence of structural components like beards, 
mustaches and glasses. 




Fig. 2: A sample set of Hajj and Umrah Facial Recognition 
Dataset (1) 

II. Pilgrims Face Recognition Dataset 

In the Hajj and Umrah face recognition dataset, HUFRD, 
hundreds of pilgrims images are taken randomly during 2011- 
2012 Hajj and Umrah seasons. The dataset contains faces from 
more than 25 countries as shown in Fig. [2] For each person, 
6 up to 15 images are taken to define a person's class in the 
dataset. Further information about this dataset can be found in 
CrowdSensing project Q. 

A. Sample Images of Hajj and Umrah Face Recognition 
Dataset 

The HUFRD contains images taken during the 2011-2012 
Hajj and Umrah seasons of a large number of pilgrims (varied 
races and appearances). It contains at least six images for each 
individual, in a varied range of poses, facial expressions (open 
and closed eyes, smiling and not smiling) and facial details 
(glasses/ no glasses), and in different lighting conditions and 
against random backgrounds. All images are in full-color JPG 
format, see Fig. [2] and 




Fig. 3: A sample set of Hajj and Umrah Facial Recognition 
Dataset (2) 



B. CrowdSensing System 

The CrowdSensing system is established to support the ex- 
isting efforts to manage the crowds and solve the missing and 
found problem during Hajj and Umrah seasons in KSA. The 
goal of this CrowdSensing system is to use techniques from 
Computer Vision and Image Processing to develop a portal 
website for Hajj and Umrah missing and found people Q. 
The system requires all pilgrims to register their personal data 
when they plan to perform Hajj or Umrah. 

One application of the proposed dataset is the CrowdSens- 
ing system, which consists of three main components, see 

Fig. a 

1) A database of all individuals arriving at the kingdom to 
perform the holy rituals. This database contains all their 
personal information along with a personal photo, and 
it can be updated via our web portal. 




Fig. 4: MFHajj portal interface website developed by the 
Crowdsensing.net team. The system consists of data collectors 
(mobiles, cameras, PCs), main server, search engine, and an 
alerting system. The system is used to recognize missing and 
found people during Hajj and Umrah seasons 



2) Advanced monitoring cameras scattered around the 
Grand Mosque in Makkah, airports, hospitals, and all 
areas of interest. 

3) Our proposed face detection & recognition algorithm is 
to be used for acquiring faces from images captured by 
the monitoring cameras and use them to identify missing 
and found individuals O. 

III. Other Face Detection and Recognition 
Datasets 

Some other face recognition research work can be found 
in 0, ED, 0, ED, d, and face recognition datasets 0, 
0, El, lfl2l . We briefly describe a note about each known 
face recognition datataset such as FAFFE, UMIST, ORL, and 
FERET. 

A. ORL Dataset 

The AT&T Database of Faces, (formerly 'The ORL 
Database of Faces'), see[5j The database was used in the con- 
text of a face recognition project carried out in collaboration 
with the Speech, Vision and Robotics Group of Engineering 
Department at Cambridge University. There are ten different 
images of each of 40 distinct subjects. For some subjects, 
the images were taken at different times, varying the lighting 
and facial expressions. All images were taken against a dark 
homogeneous background with the subjects in an upright, 
frontal position (with tolerance for some side movement). The 
images are in PGM format. The size of each image is 92x112 
pixels, with 256 grey levels per pixel (3). 

B. Japanese Dataset 

The Japanese database consists of 10 persons. The images 
per person vary from 20 to 23, where facial expressions are 
varying flD, see Fig. [6] 





P' ^3 

• -* 




d 








Fig. 5: Samples of facial images: ORL Dataset 



Fig. 8: Samples of facial images: FERET dataset 
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Fig. 6: Samples of facial images: Japanese Dataset 



C. UMIST Dataset 

The Sheffield (previously UMIST) face database consists of 
564 images of 20 individuals (mixed race/gender/appearance). 
Images are numbered consecutively as they were taken. The 
images are all in PGM format, approximately 220x220 pixels 
with 256-bit grey-scale [8], see Fig. [7] 




Fig. 7: Samples of facial images: UMIST dataset 



D. FERET 

The Face Recognition Technology (FERET) dataset is one 
of the known face recognition datasets that was collected 
between 1993 and 1996 and was developed by DARPA fl2ll . 
see Fig. [8] 

E. The Yale Face Dataset 

Yale face dataset images should be in grayscale and GIF 
format [CO. There are 11 different images of each of 15 distinct 
subjects, these different images mainly depend on different 



face expressions and little effect of light. For the same subject 
there are three different images which light is focused in 
different directions such as center, Lift, Right, and the other 8 
images related with person expression like happy, sad, sleepy, 
Shocked, with or no glasses. 




Fig. 9: Samples of facial images: Yale face dataset 



F. PIE Dataset 

PIE face dataset images in PIE Dataset are colored, it mainly 
depend on few face expressions, face pose and illumination 
condition with only one record session. There are 68 distinct 
subjects and for each subject, there are 13 different poses, 43 
different illumination with 4 different expression. lHol . 
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Fig. 10: Samples of facial images: PIE dataset 



G. Multi-PIE Dataset 

Multi-PIE face dataset images in Multi-PIE Dataset are 
colored, and mainly depend on face expressions, face pose and 
illumination condition with multiple record session. There are 
377 distinct subjects and for each subject there are 15 different 
poses, 19 different illumination with 6 different expression 
with 4 sessions. In both PIE and Multi-PIE, they support a 
3D Room with 13 camera. Also, for illumination condition, 
they support a 3D room with a flash system. Using multiple 
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Fig. 11: Samples of facial images: Multi-PIE dataset 



cameras which are fixed in specific places in the 3D room save 
time and efforts for preparing my dataset specially for large 
dataset as each person in dataset only sits in a chair with and 
fix his head. 

H. AR Face Dataset 

AR face dataset images in AR face Dataset are colored, 
and depend on face expressions such as neutral, smile, anger, 
scream, also, illumination condition (left, right, center) and 
occlusions (sun glasses and scarf) with two record sessions 
during two weeks. There are 126 distinct subjects (over 4,000 
color images) divided between man and women (70 men and 
56 women) Q. 
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Fig. 12: Samples of facial images: AR face dataset 



IV. Discussion and Conclusion 

In this note, we have described a new Hajj and Umrah face 
recognition dataset, HUFRD, for face recognition and face 
detection. During the huge overcrowds that occur in the two 
holy cities of Makkah and Madina in KSA every year (more 
than 5 millions people observe Hajj for at least 10 days), in 
addition to five millions people observe Umrah throughout 
the year. The proposed CrowdSensing system can be used 
to identify missing, found, and dead parsons. Further details 
about this system can be found in the project website El . 



TABLE I: Information about ORL, UMIIST, Japanese, 
FERET, HUFRD datasets 



dataset 


# persons 


# images 


ORL 


40 persons, 


Each has 10 images 


UMIST 


20 persons, 


vary from 19 to 48 


JAFFE 


10 persons, 


vary from 20 to 23 


FERET 


too many 


many 


HUFRD 


too many 


each has at least 6 to 15 


Yale face 


15 persons 


each has 11 images 


PIE 


68 persons 


each has 60 images 


Multi-PIE 


337 persons 


each has 40 images for one session 


AR face 


126persons 


each has 26 images 
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Fig. 13: Pilgrims facial dataset (HUFRD) for face detection and recognition 



